Repository navigation
Keep the worker alive through transient runtime failures while it registers (sdk-php#448 parity) - #355
Merged
Conversation
Same issue as durable-workflow/sdk-php#448. Worker.run() and run_until() raised once the client's three transport attempts failed on cluster discovery or registration, so a native Server answering 503 {"reason":"storage_unavailable"} during warm-up (or a refused connection, or an upstream 502/503/504) ended the worker. Discovery and registration are now repeated whole through transient failures with the shared backoff uniform(0, min(4, 0.25 * 2**(n-1))) s: refused/reset/timed-out connections, upstream 502/503/504 without a protocol body, 503 storage_unavailable/backend_unavailable, and 5xx/429 with retryable: true. retryable: false, 401/403, other 4xx and an incompatible Server stay fatal and are logged at error level before run() raises. Pollers, completions and worker heartbeats already rode out these failures and are unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Coordinator decisions for #355: - An activity completion or failure report that hits a transient is resent with the shared backoff uniform(0, min(4, 0.25 * 2**(n-1))) s while the attempt's lease (renewed lease, else the claimed lease_expires_at) covers the next wait. A transient never becomes a failure report. When the lease runs out the resend stops with an error log and the Server settles the attempt at lease expiry. - A resend answered 409 stale_attempt, recorded false, same task/attempt /owner, activity/attempt/task status all completed is this worker's committed completion (observed on durableworkflow/server:3.17.0; same rule as PHP's isCommittedActivityCompletion). Any other stale_attempt is a closed attempt, never success. - Poll-time 401/403 keep retrying so credentials can rotate, but every refusal is now logged at ERROR. The cross-SDK difference (PHP and Rust stop) is documented. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…RROR Review follow-ups for #355: - Poll retry waits are uniform(0, min(10, 0.25 * 2**(n-1))) s (the shared full jitter, capped at 10 s instead of growing to 30 s), still honouring a bounded Server retry delay. - A poller failing continuously for 60 s logs the outage at ERROR, then every 5 minutes while it lasts; a successful poll resets it. Co-Authored-By: Claude Opus 5.5 <[email protected]>
rmcdaniel
added a commit
that referenced
this pull request
Nov 20, 2026
The duplicate-completion check runs when the committed-completion rule (#355) is present; it logs at INFO, so the test captures that level. Verified against this branch merged with #355. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Keeps both: this branch's transient retries, committed-completion rule and outage logs, and main's decoded response cap and continue-as-new result. Only CHANGELOG.md conflicted; both entries are kept. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
rmcdaniel
added a commit
that referenced
this pull request
Nov 20, 2026
Brings in the committed-completion rule, so the end-to-end test's garbled completion resend (409 stale_attempt) is accepted as success. Only CHANGELOG.md conflicted; both entries are kept. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Python counterpart of durable-workflow/sdk-php#448: a worker process died when native Server answered
503 {"reason":"storage_unavailable","message":"storage_unavailable"}during warm-up.What the audit found (origin/main
cd5cff4)I drove
Worker.run()against a mock transport that answers503 storage_unavailableon one route ten times and then recovers. These are the results on main:GET /api/cluster/infoat startupWorker._registerwraps the error asRuntimeError("Server compatibility error: unable to read /api/cluster/info: server returned 503 ...")once the client's 3 transport attempts (about 0.3 s) are used up.POST /api/worker/registerat startupServerError(503 storage_unavailable)._poll_*_taskscatch every exception and back off with jitter (_PollFailureBackoff); only storage-admission errors are raised._send_heartbeatlogs the failure and returnsNone.Change
Cluster discovery and registration are now repeated whole through transient failures, in both
run()andrun_until()(within its timeout). The wait follows the shared scheduleuniform(0, min(4, 0.25·2^(n−1)))seconds, each retry logs a warning, and shutdown interrupts the wait.Transient means:
storage_unavailableorbackend_unavailableretryable: trueStill fatal:
retryable: falseFor these, the worker logs
worker <id> stopping: registration failed with a non-retryable error: ...at error level, andrun()raises._registeritself is unchanged, so the replay-guarded symbol is untouched. Pollers, heartbeats and completions are unchanged. Docs: a new "Startup while the runtime is unavailable" section indocs/reference/worker.md.One difference from the PHP rule: after registration, Python pollers keep retrying 401/403. See the follow-up below.
Follow-up: activity outcomes and poll auth refusals (coordinator decisions)
Activity outcomes. On main, a completion or failure report that hit a transient was retried only by the client's 3 quick attempts, then dropped with a warning (
complete_error).lease_expires_atotherwise.activity task <id> completed outcome not delivered after N transient failures (...): its lease ran outat error level and stops. The Server settles the attempt at lease expiry.Duplicate completion, observed on the published
durableworkflow/server:3.17.0. I put an in-process transport in front of a real worker that forwards the first/completeand then shows the client a 503 or a connection reset. The Server answers the resend with:isCommittedActivityCompletion. With this branch, the run completes with exactly oneActivityCompleted, the activity ran once, and the worker keeps running. That held for both the 503 and the reset cases.stale_attemptis a closed attempt and is never treated as success.409 {"outcome":"failed","recorded":false,"reason":"stale_attempt"}, with no owner or status fields. It is reported as a closed attempt. History still has exactly oneActivityFailed.Poll-time 401/403. Pollers still retry so rotated credentials recover without a restart; that behavior is documented. Every refusal is now logged at ERROR, not rate-limited, including refusals before a later recovery. The difference from PHP and Rust, which stop the worker, is documented in
docs/reference/worker.md.Review follow-ups: poll backoff and outage logging
uniform(0, min(10, 0.25·2^(n−1)))seconds. That is the shared full jitter capped at 10 s, instead of a ceiling that doubled from 1 s to 30 s. A bounded Server retry delay (minimum_delay) is still honoured.tests/test_poll_failure_backoff.py):0.25, 0.5, 1, 2, 4, 8, 10, 10, 10, reset after a success(0, ceiling)minimum_delayis honoureddocs/reference/worker.mdis updated.Tests (
tests/test_worker_transient_runtime.py)Added for the follow-up:
stale_attemptshapes are not success, and none of them is resent.tests/test_poll_failure_backoff.py: every 401/403 is logged at ERROR, including before a recovery.From the first round:
storage_unavailableandbackend_unavailable,retryabletrue and false,storage_pressure, 401, 403, 409, and compatibility errors.ClientandWorker.run()overhttpx.MockTransport. Eight transient failures on discovery or registration (503storage_unavailable, refused connection, bare 502,backend_unavailable), and two on the activity poll, are followed by recovery. The worker keeps running, registers, polls, and logs nothing at error level.503 retryable: false. Each is served once, never retried, never polls, and logs the error-level stop line.Locally (Python 3.14):
pytest tests/ -m "not integration": 7855 passed, 3 skipped.🤖 Generated with Claude Code